We thank all reviewers for their critical comments and we address some questions below. Y es, we have tried the two-stage approach in our initial study. Similar as what many existing works reported [e.g. 1, MGDA is more robust and parameter-free. We did not perform statistical test since some baselines only used the default train/test split on MNIST. Our setup here also comply with existing works [e.g. 1, 5, 16, 34, 38, 39, 51, 52].

Machine learning models can make critical errors that are easily hidden within vast amounts of data. Such errors often run counter to rules based on human intuition. However, rules based on human knowledge are challenging to scale or to even formalize. We thereby seek to infer statistical rules from the data and quantify the extent to which a model has learned them. We propose a framework SQRL that integrates logic-based methods with statistical inference to derive these rules from a model's training data without supervision. We further show how to adapt models at test time to reduce rule violations and produce more coherent predictions. SQRL generates up to 300K rules over datasets from vision, tabular, and language settings. We uncover up to 158K violations of those rules by state-of-the-art models for classification, object detection, and data imputation. Test-time adaptation reduces these violations by up to 68.7% with relative performance improvement up to 32%. SQRL is available at https://github.com/DebugML/sqrl.